BACKGROUND

Persons with cognitive disabilities, resulting from conditions such as brain injury, often require rehabilitation services and long-term support for successful community reintegration, including employment.  Research has demonstrated that some of the more effective approaches for persons with cognitive disabilities involve the application of specific strategies, including accommodations and supports, occurring in the natural environment where obstacles are encountered that impact performance. 

In vocational rehabilitation one such intervention that has been effective with this population is supported employment (SE).   SE provides individualized services to an individual with a disability in the workplace through a job coach who assists the client in maintaining satisfactory job performance (1, 2).  Job coaches often address cognitive problems that can impact vocational performance by monitoring the client as he or she performs job related tasks and providing cuing responses to address problems with initiation, attention, memory, learning and sequencing. In order for this SE approach to be effective, job coaching services are provided in person in the natural employment setting and often require full-time, one-to-one intervention for an indefinite period of time. While there is support of job coaching as an effective intervention for persons with traumatic brain injury (TBI), commitment of staff and financial resources can be very challenging for vocational rehabilitation providers (3).  As a result, consistent provision of job coaching services is often not readily available to rehabilitation consumers to be provided on the “as needed” basis that is recommended. Funding can tend to steer programming affecting the intensity, duration and availability of job coaching services (4, 5, 6).Another problem that has been  identified in placing a job coaches the work environment is that it can draw unwanted attention to the consumer (and the disability) and may not be readily accepted by other persons in the environment (for example, an employer or co-worker).

A telerehabilitation-based “in vivo” intervention is being developed to support consumers who have cognitive disabilities.We are implementing activity recognition using accelerometers to monitor specific pre-determined work behaviors and, ultimately, to deliver cues and instruction in response to problem behaviors. The system is trained using video labeling synchronized with data regarding motion from accelerometers worn while performing a given vocational task being performed correctly, as well as for common errors.  A model is then developed to automatically recognize components of the task (or errors) and provide feedback to a client as appropriate (e.g., a task sequence error triggers a task instruction delivered via headphone to repeat the step in the proper sequence). A diagram of the system is shown in Figure 1.

The goals of the project are as follows:

1. To use motion technology to ascertain task specific, qualitative behavioral data regarding individuals with cognitive/behavioral disabilities performing job-related tasks.
2. To determine the reliability and validity of this technology for “in vivo” behavioral assessment.
3. To use the capacity of the technology to deliver specific task instructions (job coaching) based upon the behavioral assessment described above.
4. To determine the utility (validity, cost-effectiveness) of the technology for enhancing performance of specific work tasks in persons with significant cognitive disabilities.
5. To identify the specific environmental and client characteristics that indicate that such technology is likely to be effective.      

PROGRESS

A series of focus groups and interviews with employers and vocational rehabilitation providers, and observations at industrial and food service work sites were conducted. Through this collaboration we identified food preparation as a domain in which many people requiring job coaching are employed and which has tasks for which this technology might be appropriate.  More specifically, we identified the task of grilling hamburgers in a fast food setting as an initial application. In institutional settings this task requires a long series of actions which must be performed in proper sequence and pace to create a high quality product.

This hamburger grilling task was initially replicated in a motion capture lab at Carnegie Mellon University.  Video was captured for an able-bodied subject performing the task.  A machine learning technique (AdaBoost) was then used to automatically identify features in the person’s movement patterns by which the system could automatically recognize components of the task (e.g. flipping burgers, salting burgers, placing burgers on or taking burgers off the grill).

Performance of the model is shown in Table 1, analyzing the testing data for a model based on the training data.

Table 1: Performance for the AdaBoost Model

Action	Total Actions	False Positives	False Negatives	Merged Actions	Separated Actions
Flipping	672	1 (0.15%)	0	18 (2.6%)	2 (0.29%)
Picking	224	0	0	0	3 (1.3%)
Placing	224	0	0	0	0
Pressing	444	0	0	0	0
Salting	448	0	0	38 (8.5%)	1 (0.22%)

None of the actions were mislabeled.  There was one erroneously labeled flip, but it appears this was a flip which was aborted when the subject realized he'd forgotten to press the burgers. The most common errors are actions which are merged or separated.  Merged actions occur when actions are performed with very little time between them and actions are split when there the subject appears to stop mid-action.

The set-up has since been relocated to a lab at the University of Pittsburgh as we further train and refine the system with additional able-bodied subjects and begin to apply this approach with a small population of persons with TBI in a pilot study.

The performance of the model has been tested on additional able-bodied subjects and compared to traditional job coaching observations. IRB approval has been obtained and subjects are currently being recruited to determine the efficacy of the system as applied to a clinical population of persons with traumatic brain injury.

DISCUSSION

 Initial work has yielded a model which is able to correctly identify component tasks for multiple subjects without disabilities.  Next steps include (1) testing the model’s ability to recognize movement patterns for additional subjects; (2) testing the model’s ability to recognize movement patterns for individuals with cognitive impairments; (3) processing the activity recognition output to determining when the task is being performed correctly and when errors are being made (e.g. pressing should follow salting, otherwise there’s an error); (4) develop a user interface to prompt the client when errors are made (or to provide occasional praise); and (6) develop an interface to share a summary of the data (e.g. success rate, types of errors) with a job coach or other support person.

Once these tasks are accomplished, the complete system will be tested for a larger sample of people with cognitive impairments, in which the system will observe their performance of the task and provide cues as needed.  Data will include the match between the system’s identification of task errors and the true incidence of task errors, as well as the number of uncorrected errors with and without the intervention.  This evaluation will be used to determine whether the system is effective in assisting people with disabilities, as well as the settings in which it is likely to be effective; for example, whether the system has promise as a cueing system on the final job site or is more likely to be effective as a training tool.